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(54) Rim grain matching in composite images 



(57) Image data is processed by identifying a region of 
substantially constant colour within a first image. Colour 
variations, in terms of standard deviation about an 
average colour, are analysed and a similar level of 
variation is applied to a second image. The first image 
may be derived from cinematographic film, with colour 
variations due to film grain. The second image may be a 
video image or a computer generated image and the 
application of a similar level of variation may reproduce 
noise similar to that present within the film due to grain. In 
this way, it is possible to add grain to non-filmed images 
so as to match said images to images derived from film. 
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PROCESSING IMAGE DATA 

5 The present invention relates to processing image data in order to 

facilitate composting and editing. 

Introduction 

The processing of broadcast quality video signals in the digital domain 
10 has become increasingly popular and has created a significant number of 
new production techniques. By using fast digital frame stores and parallel 
disk drives it is possible to produce full bandwidth digital video signals in real- 
time, thereby facilitating on-line video editing and compositing. This has led to 
a demand for such techniques to be used in cinematographic film editing, 
1 5 although the editing of cinematographic film is significantly more difficult given 
the relatively high information content of film compared to broadcast quality 
video. 

In order for images conveyed by film media to be processed using 
digital techniques, it is necessary to scan the image frames to produce 

20 manipulatable digital image signals. After manipulation, these digital signals 
are supplied to an exposing device, arranged to expose destination film 
stock, thereby returning the manipulated image back onto a film medium. 
From the film viewer's point of view, a manipulated film image should be 
indistinguishable from an unmanipulated film image. 

25 One area in which there is a problem of being able to distinguish 

between a manipulated film image and an unmanipulated film image, is 
where a film clip comprises a plurality of film image frames, each made up of 
a film image and a video or computer generated image. For example, where 
a video image or computer generated image of a foreground feature is 

30 composited with a filmed image of a background feature, there may be a 
mismatch in the textural qualities between the foreground image and the 
background image due to the "graininess" of the film image. The video or 
computer generated portion of the composite image can lack graininess or 
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noise, or have a different noise characteristic to the portion of the composite 
clip which originated as a filmed image. 

The graininess of a film is a visual sensation experienced by a viewer 
of the film, in which the viewer experiences a subjective impression of a 
5 random dot-like textural pattern in an image. When an image is projected 
onto a large area screen, the graininess of a filmed image may be readily 
apparent 

Graininess in a viewed film clip results from the physical 
composition of the film itself. Motion picture films consist of silver halide 

10 crystals dispersed in a gelatine emulsion. The exposure and development of 
the crystals form the photographic image, which is made up of the discrete 
particles of silver. In color processes, the silver is removed after development 
of the film, and dyes form dye clouds centred on sites of developed silver 
crystals. The crystals can vary in size, shape and sensitivity to light, and are 

1 5 generally randomly distributed within the emulsion. 

Individual silver particles can range from about 0.002 mm, down to 
about 0.0002 mm. In a motion picture, the eye cannot distinguish individual 
grain particles, however the particles resolve into random groupings of dense 
and less dense areas which, when viewed, result in the visual sensation of 

20 graininess. 

Figure 1 of the accompanying drawings illustrates clouds of dye 
formed at sites occupied by exposed silver halide. Figure 2 of the 
accompanying drawings shows discrete grains at a magnification of around 
400x. Grains nearer the surface of the film are in focus, whilst grains deeper 

25 in the emulsion are out of focus. Figure 3 of the accompanying drawings 
shows the make up of individual grains of filamentary silver enlarged by an 
electron microscope. 

It is known for suppliers of particular film types to produce sets of 
curves characterizing the graininess of a batch or type of film in terms of the 

30 "granularity" of the film. A statistical measure of the granularity of a sample 
may be supplied by the film manufacturers in the form of an RMS granularity 
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characteristic measured by an electro-optical microdensrtometer. 

However, an RMS granularity characteristic of a batch of film, or film 
type, may become inaccurate, due to factors such as temperature, the 
ageing of film, exposure levels and other factors. A knowledge of the RMS 
5 granularity of the film on which a particular film clip is recorded will often not 
give an accurate enough basis for matching the granularity of an individual 
film clip. 

Summary of the Invention 

10 According to the first aspect of the present invention, there has 

provided a method of processing image data, comprising steps of identifying 
a region of substantially constant colour within a first image; analysing colour 
variation within said region; and applying a similar level of variation to a 
second image. 

15 Preferably, a plurality of regions are selected, wherein each of said 

regions has a different colour In a preferred embodiment, mean or average 

values are calculated for the intensities. 

In a preferred embodiment, standard deviation values are calculated 

for the intensities and these values may be considered as a function with 
20 respect to colour. Preferably, the step of applying a similar level of variation 

to a second image involves applying a similar standard deviation to areas 

within said second image. 

According to a second aspect of the present invention, there is 

provided an image data processing apparatus, comprising identification 
25 means configured to identify a region of substantially constant colour within a 

first image; analysing means configured to analyse colour variation within 

said region; and applying means configured to apply a similar level of 

variation to a second image. 

In a preferred embodiment the identification means is configured to 
30 identify a plurality of regions of differing colours and said analysing means 

may be arranged to calculate mean or average values for said colour 
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variations. Preferably, the analysing means is configured to calculate 
standard deviation values and said deviation values may be calculated with 
respect to each colour. 

In a preferred embodiment, the analysing means analyses standard 
5 deviation values of said first image and said applying means is configured to 
apply similar degrees of standard deviation to said second image. 

Means may be provided for deriving the first image from 
cinematographic film. Means may be provided for deriving said second image 
from video data or, alternatively, a computer may be configured to generate 
10 said second image. 

In a preferred embodiment, the analysing means is configured to 
analyse a plurality of regions by traversing linearly across the first image. 

Brief Description of the Drawings 

15 Figures 1 to 3 show examples of granularity in cinematographic film; 

Figure 4 shows a general overview of a compositing process; 
Figure 5 shows an image processing apparatus adapted for 
compositing a video image or computer generated image with a film image 
and applying an appropriate granularity texture to the composite image; 
20 Figure 6 shows a graphics work station comprising the image 

processing apparatus; 

Figure 7 shows an over view of a process for applying a grain texture 
in a composite image; 

Figure 8 shows a background film image; 
25 Figure 9 shows a foreground video image or computer generated 

image; 

Figure 10 shows a composite image of the background and 
foreground images of Figures 8 and 9; 

Figure 11 shows the composite image of Figure 10, having applied 
30 thereto a grain texture; 

Figure 12 shows a method of scanning the film background image of 
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Figure 8; 

Figure 13 shows a method of sampling and characterizing a 
granularity of a portion of the filmed image of Figure 12; 

Figure 14 shows schematically a process for characterizing a 
5 granularity of a region of a filmed image; 

Figure 15 shows a spatial intensity distribution of a region of the filmed 

image; 

Figure 16 shows a gradient of the spatial intensity distribution function 
of Figure 15; 

10 Figure 17 shows schematically characterization of granularity of first 

and second regions of the filmed image of Figure 8; 

Figure 18 shows schematically a process for production of granularity 
characteristics from a plurality of spatial intensity distributions sampled at 
various regions on the filmed image of Figure 8; 
15 Figure 19 shows a granularity characteristic determined in accordance 

with a specific method according to the present invention; 

Figure 20 shows a general process for synthesization of a granularity 
template signal for a video image or computer generated image; and 

Figure 21 shows schematically an apparatus for synthesization of the 
20 granularity template signal for a video image or computer generated image. 

Detailed Description of the Preferred Embodiments 

Referring to Figure 4 of the accompanying drawings, there is shown 
an overview of a general preferred process for compositing film image data 

25 with video image data or computer generated image data. The filmed image 
is converted to a first digital image signal corresponding to the film image 
data, and the first digital image signal is entered into a digital editing suite 
100. A second digital signal corresponding to a video image or computer 
generated image, generated by a video source or computer source 101, is 

30 input to the editing suite 100, and the first digital image signal is composited 
with the second digital image signal to produce a composite digital image 
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signal. The composite video image signal is converted to a film image on a 
reel of film by exposing the film. 

Referring to Figure 5 of the accompanying drawings, there is shown 
apparatus for performing the process as described above. The first digital 
5 image signal may be produced by a film scanner 501 and passed along a 
wideband data bus 502 for storage in a high capacity storage device 504. 
The second digital image signal, corresponding to a video image or a 
computer generated image may be input from a video signal source device or 
computer signal source device attached to the data bus 502. The first digital 

10 image signal and second digital image signal may be composited in a 
graphics processor 505, linked to a graphics workstation 506 comprising a 
screen 507, a control keypad 508, and a tablet and pen 509. A resulting 
composite digital image signal may be downloaded and converted back to a 
film storage medium by a film recorder 510 receiving the composite digital 

1 5 image signal from the data bus 502. 

In the present specification rt will be understood that the signals carry 
data corresponding to images and the data may be converted to signal form, 
for storage purposes and data processing operations. Selection or 
identification of a portion of a displayed image may correspond to selection or 

20 identification of the corresponding data for that portion. 

Figure 6 shows schematically an editing artist in overall control of the 
compositing process at the graphics work-station 506. 

Referring to Figure 7 of the accompanying drawings there is shown a 
background image frame on film, the background image being subject to a 

25 spatial intensity texture or granularity, which as seen by a viewer would be 
described as a "graininess" of the image. 

Referring to Figure 8 of the accompanying drawings, there is shown a 
video image frame or computer generated image frame, having no graininess 
as seen by the observer. 

30 Shown in Figure 9 of the accompanying drawings, is a composite 

image comprising the background image of Figure 7 and the foreground 



7 



image of Figure 8. In the composite image, the observer perceives a 
difference in the graininess of the foreground object and the background. The 
difference in graininess between foreground and background objects is 
undesirable in the composite image since the human observer perceives the 
5 composite image to be unrealistic. 

Referring to Figure 10 herein, there is described an overview of a 
process for producing a composite image data derived from filmed image 
data or signal and a video or computer generated image or signal, in which 
the video or computer generated portion of the composite image data or 
10 signal has applied thereto a textural pattern data or signal synthetically 
generated to match the grain texture of the filmed background image. 

Figure 11 herein illustrates the resultant composite image having 
applied thereto the synthetically generated grain texture. 

There will now be described a method of characterizing a grain texture 
15 of a filmed image, with reference to Figures 12 to 19 of the accompanying 
drawings. 

A frame of filmed image is converted into a first digital image signal as 
described above. The first digital image signal may be a video signal. The 
first digital image signal is displayed upon a display device, in this case the 

20 display unit 507 of the graphics work-station. The first digital image as 
displayed, is illustrated schematically with reference to Figure 12 herein. 

An operator of the graphics work-station, eg a film editing artist, using 
an electronic cursor and the tablet and stylus 509, identifies a region of the 
displayed image in which there is a relatively uniform region, for example the 

25 region 1 in Figure 12. By relatively uniform it is meant relatively uniform with 
respect to overall color. 

Referring to the upper portion of Figure 13 herein, there is shown the 
region 1 of Figure 12 in magnified form. The region 1 of the image displayed 
on the display unit corresponds to a corresponding region of the original film 

30 image contained on the film. The granularity of the original film of the film clip 
causes fluctuations in the intensity of the portion of the first digital image 



signal corresponding to the displayed region of relatively uniform color 1 . On 
selecting a region of relatively uniform color, the assumption is made that any 
variations in the intensity across region 1 are caused by the granularity of the 
original film frame corresponding to region 1 . 
5 Once the operator has identified relatively uniform region 1, the 

graphics processor 105 operates to scan the region of relative uniformity, for 
example along a straight line between spatially disparate points A and B as 
shown in Figure 13. As the graphics processor 505 scans the corresponding 
region of the first digital image signal, corresponding to the spatially disparate 
10 positions A and B on the displayed image, the processor records values of 
intensity against distance. Where the first digital image signal is a video 
signal, the processor may record variations in intensity on a pixel by pixel 
basis. 

The portion of the first digital image signal corresponding to the line 

15 AB in the region 1 may be filtered into separate color components 
corresponding to red, green and blue components. For each of the red, 
green and blue components, a characteristic of intensity versus distance may 
be recorded and stored. The resultant plot of intensity versus distance, for the 
purposes of convenience in this specification shall be referred to as a spatial 

20 intensity distribution. 

The spatial intensity distribution is then further processed as follows. 
It has been found that the variation of intensity with distance over a 
region of a filmed image fluctuates in accordance with the granularity of the 
film. To characterize the granularity of the film the spatial intensity distribution 

25 from the line A-B of the region 1 of the first digital image signal is firstly 
averaged by computer program applying an algorithm to the data of the 
spatial intensity distribution, to produce a mean or average value mu. The 
standard deviation of intensity as it varies above and below the mean or 
average intensity mu is determined. For example the standard deviation 

30 sigma may be determined in accordance with the formula shown in Figure 
12, where: 
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sigma = standard deviation of intensity. 

mu = mean or average intensity over the region AB 

K = a constant, and 

lambda = frequency of intensity fluctuations. 



Secondly, by determining the gradient of intensity fluctuations with 
distance over the line A-B, a periodicity P of intensity fluctuations around the 
mean or average value mu may be determined from the gradient 

10 characteristic. The periodicity P is dependent upon the grain size in the 
original filmed image. 

Thus, for the region 1, sampled along the line A-B, information about 
the granularity of the original film may be determined and characterised in 
terms of the parameter's standard deviation, mean or average intensity (mu), 

15 and periodicity P of the gradient of intensity change. This data gives 
information about the granularity of the film used in the film clip from a small 
relatively uniform sample region 1 of one frame of the film clip at one overall 
color. 

However, the film clip comprises a large number of image frames, and 
20 the color varies over each image frame. A general objective in characterizing 
the grain texture of the film used for the film clip is to select specific regions of 
relative uniformity. Since, in the absence of grain structure in the film, 
individual regions of relatively constant overall color ought not to show any 
variations in color intensity in either of the three red, green or blue 
25 components, any variations in intensity determined from the first digital image 
signal portions corresponding to the regions of relative uniformity ought to be 
due solely to the granularity of the originating film. 

To obtain a better characterization of the grain texture of the 
originating film, the steps shown in Figure 13 are repeated for a number of 
30 different regions of relative uniformity. The different regions of relative 
uniformity should be of different colors. Each region of relative uniformity of 
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different color will have its own particular value of mean or average intensity 
mu for the red, green and blue components. 

The steps are summarized in schematic form in Figure 14. 
Referring to Figure 15 of the accompanying drawings, there are 
5 shown first and second regions of relative uniformity 1, 2 which are 
respectively sampled along lines A-B and C-D. Each sampling results in 
corresponding spatial intensity distributions along lines A-B and C-D in the 
red, green and blue components. There may be determined by a plurality of 
standard deviations sigma r1, sigma g1, sigma b1, sigma r2, sigma g2, sigma 

10 b2 corresponding to the spatial intensity distributions of the first and second 
regions. Similarly, for each spatial intensity distribution there may be 
determined a particular value of P relating to the grain size. 

Referring to Figures 12 to 17 of the accompanying drawings, where a 
number N of individual regions of relative uniformity are sampled, there may 

1 5 result in a number N of values mu, mu 1 ... mu M. For each value of mu, there 
will result a corresponding value of sigma. Values of mu and sigma may be 
plotted as shown in Figure 17 with mu on the horizontal axis and sigma on 
the vertical axis, to produce a color plane granularity characterization map 
which characterizes the granularity of the originating film by finding discrete 

20 points of standard deviation at discrete values of mean or average color 
intensity mu. 

By sampling N regions, a continuous curve for each of the red, green 
and blue components may be approximated by fitting a polynomial 
expression to the determined values of sigma for the respective red, green 
25 and blue components. 

Referring to Figures 18 and 19 of the accompanying drawings, there is 
shown graphically, a preferred method for determining the grain size P from 
the spatial intensity distributions. 

Figure 18 shows a spatial intensity distribution of, for example, a red 
30 component. Figure 19 shows the gradient of the spatial intensity distribution 
of Figure 18. A periodicity P is determined as a most prominently present 
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fundamental frequency of the gradient of the intensity, over the sample 
region A-D. 

A preferred method of synthetically generating a granularity template 
signal or data for applying to the second digital image signal corresponding to 
5 the video image or the computer generated image, will now be described. 

Referring to Figure 20 of the accompanying drawings, a noise 
generator 600 generates signal intensity noise to mimic the intensity 
fluctuations in the first digital image signal produced by the granularity of the 
film. A low pass filter 601 filters the noise signal generated by the noise 
10 generator 600. 

The noise generator 600 produces noise having a standard deviation 
which can be specified by inputting a standard deviation value sigma. The 
low pass filter can filter the random noise signal generated by the noise 
generator 600 with a cut off frequency determined by the periodicity P, which 
1 5 characterizes the grain size of the originating film clip. 

In the composite signal, comprising the first digital image signal and 
the second digital image signal, the first digital image signal contains 
information relating to the granularity of the originating film. By applying the 
granularity template signal produced by the random noise generator 600 and 
20 low pass filter 601, the granularity template signal is produced having 
characteristics which relate to the granularity of the film. This is then used to 
produce an effect on the second digital image signal (the signal 
corresponding to the video image or to the computer generated image) 
which mimics the granularity of the originating film when an image 
25 corresponding to the composite digital image signal is displayed. 

When the composite digital image signal is transferred back to film 
medium via the film recorder 510, the resultant film holds an image 
comprising the image on the original originating film clip plus the keyed-in 
video image or computer generated image, in which the keyed-in portion of 
30 the composite image contains a granularity effect comparable to that on the 
originating film. 
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Claims 

1. A method of processing image data, comprising steps of 
identifying a region of substantially constant colour within a first image; 

5 analysing colour variation within said region; and 

applying a similar level of variation to a second image. 

2. A method according to claim 1, wherein a plurality of regions 
are selected, wherein each of said regions has a different colour. 

10 

3. A method according to claim 1 or claim 2, wherein mean or 
average values are calculated for said intensities. 

4. A method according to claim 3, wherein standard deviation 
1 5 values are calculated for said intensities. 

5. A method according to claim 4, wherein said standard 
deviations are considered as a function with respect to colour. 

20 6. A method according to any of claims 1 to 5, wherein said step 

of applying a similar level of variation to a second image, involves applying a 
similar standard deviation to said image. 

7. A method according to any of claims 1 to 6, wherein said first 
25 image is derived from cinematographic film. 

8. A method according to any of claims 1 to 7, wherein said 
second image is derived from a video. 

30 9. A method according to any of claims 1 to 7, wherein said 

second image is computer generated. 
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10. A method according to any of claims 1 to 9, wherein a plurality 
of regions are analysing by traversing linearly across said image. 

11. Apparatus processing Image data processing apparatus, 
comprising identification means configured to identify a region of substantially 
constant colour within a first image; 

analysing means configured to analyse colour variation within said 
region; and 

applying means configured to apply a similar level of variation to a 
second image. 

12. Apparatus according to claim 1 1 , wherein said identification 
means is configured to identify a plurality of regions of differing colours. 

1 3. Apparatus according to claim 1 1 or claim 12, wherein said 
analysing means is arranged to calculate mean or average values for said 
colour variations. 

14. Apparatus according to claim 13, wherein said analysing 
means is configured to calculate standard deviation values. 

15. Apparatus according to claim 14, wherein said analysing 
means considers standard deviation values with respect to colour. 

1 6. Apparatus according to any of claims 1 1 to 1 5, wherein said 
analysing means analyses standard deviation values of said first image and 
said applying means is configured to apply similar degrees of standard 
deviation to said second image. 



14 

17. Apparatus according to any of claims 1 1 to 16, including means 
for deriving of said first image from cinematagraphic film. 

18. Apparatus according to any of claims 1 1 to 17, including means 
5 for deriving said second image from video data. 

19. Apparatus according to any of claims 1 1 to 17, including a 
computer configured to generate said second image. 

1 0 20. Apparatus according to any of claims 1 1 to 1 9, wherein said 

analysing means is configured to analyse a plurality of regions by traversing 
linearly across said first image. 

21 . A method of processing image data substantially as hear in 
1 5 described with reference to figures 4 to 21 . 

22. Image processing apparatus substantially as hear in described 
with reference to figures 4 to 21 . 
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